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Abstract 


He  describe  a  program  for  verifying  that  a  set  of  rules  in  an  expert  system 
comprehensively  spans  the  knowledge  of  a  specialized  domain.  The  program  has 
been  devised  and  tested  within  the  context  of  the  ONCOCIN  System,  a  rule- 
based  consultant  for  clinical  oncology.  The  stylized  format  of  ONCOCIN’ s 
rules  has  allowed  the  automatic  detection  of  a  nunber  of  common  errors  as  the 
knowledge  base  has  been  developed.  This  capability  suggests  a  general 
mechanism  for  correcting  many  problems  with  knowledge  base  completeness  and 
consistency  before  they  can  cause  performance  errors. 
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1  Introduction 

The  builders  of  a  knowledge-based  expert  system  must  ensure  that  the 
system  will  give  Its  users  accurate  advioe  or  correct  solutions  to  their 
problems.  The  process  of  verifying  that  a  system  is  accurate  and  reliable 
has  two  distinct  components:  checking  that  the  knowledge  base  contains  all 
necessary  information)  and  verifying  that  the  program  can  interpret  and  apply 
this  Information  correctly.  The  first  of  these  components  has  been  the  focus 
of  the  current  research ;  the  second  corresponds  to  the  familiar  problem  of 
program  "debugging"  and  will  not  be  discussed  in  this  paper. 

Knowledge-base  debugging,  the  process  of  checking  that  a  knowledge 
base  is  correct  and  complete t  is  one  component  of  the  larger  problem  of 
knowledge  acquisition.  This  process  Involves  testing  and  refining  the 
system's  knowledge  in  order  to  discover  and  eorrect  a  variety  of  errors  that 
can  arise  during  the  process  of  transferring  expertise  from  a  human  expert  to 

4 

a  computer  system.  In  this  paper,  we  discuss  some  common  problems  in 
knowledge  acquisition  and  debugging,  and  describe  an  automated  assistant  for 
checking  the  completeness  and  consistency  of  the  knowledge  base  in  the 
ONCOCIN  system  [31. 


2  Knowledge  Acquisition 

Before  knowledge  oan  be  embodied  in  a  computer  system,  it  must 
undergo  a  ntmber  of  transformations.  First,  a  human  acquires  expertise  in 
some  domain  through  study,  research  and  experience.  Next,  the  expert 
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attempts  to  formalize  this  expertise  and  to  express  it  in  the  internal 
representation  of  an  expert  system*  e.g.,  production  rules,  frames,  or 
semantic  nets.  Finally,  the  knowledge,  in  a  machine- readable  form  such  as 
LISP  expressions,  is  added  to  the  computer  system's  knowledge  base. 

Problems  can  arise  at  any  stage  in  this  process:  the  expert's 
knowledge  may  be  incomplete,  inconsistent,  or  even  partly  erroneous. 
Alternatively,  accurate  and  complete  knowledge  may  not  be  adequately 
transferred  to  the  computer-based  representation.  The  latter  problem 
typically  occurs  when  an  expert  who  does  not  understand  computers  works  with 
a  knowledge  engineer  who  in  unfamiliar  with  the  problem  domain; 
misunderstandings  that  arise  are  often  unrecognized  until  performance  errors 
occur.  Finally,  spelling  or  syntax  mistakes  that  are  made  when  the 
knowledge-base  is  entered  into  the  computer  are  a  frequent  source  of  errors. 


3  Why  an  Automated  Assistant  for  Khowledge-base  Debugging? 

The  knowledge  base  of  an  expert  system  is  generally  constructed 
through  collaboration  between  experts  in  the  problem  domain  and  knowledge 
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engineers.  The  domain  experts  formulate  their  knowledge  and  the  knowledge 
engineers  encode  this  knowledge  for  use  by  the  system.  This  difficult  and 
time-consuming  task  can  be  facilitated  by  a  program  which: 

(1)  checks  for  inconsistencies  and  gaps  in  the  knowledge  base, 


(2)  helps  the  experts  and  knowledge  engineers  to  communicate  with  each 
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(3)  provides  a  clear  and  understandable  display  of  the  knowledge  as  the 
system  will  use  it. 

An  autoaated  assistant  for  the  systea  builders  oould  rapidly  identify 
problems  in  the  system's  knowledge  base  and  possibly  allow  the  experts  to 
discover  gaps  in  their  knowledge  or  errors  in  their  reasoning. 


4  Knowledge-Base  Debugging 


4.1  Earlier  Work 

One  goal  of  the  TEIRESIAS  program  [1]  was  to  automate  knowledge-base 
debugging  in  the  context  of  the  MIC IN  infectious  disease  consultation  system 
[2].  TEIRESIAS  allowed  an  expert  to  judge  whether  MTCIN's  diagnosis  was 
correct,  to  track  down  the  errors  in  the  knowledge  base  that  led  to  incorrect 
conclusions,  and  to  alter,  delete  or  add  rules  in  order  to  fix  these  errors. 
The  knowledge  transfer  occurred  in  the  setting  of  a  problem-solving  session; 
no  formal  assessment  of  rules  occurred  at  the  time  they  were  initially 
entered  into  the  knowledge  base. 

In  the  EMTCIN  system  for  building  knowledge-based  consultants  [4], 
the  knowledge-acquisition  program  fixes  spelling  errors,  checks  that  rules 
are  semantically  and  syntactically  correct,  and  points  out  potential 
erroneous  Interactions  among  rules.  In  addition,  ENTCIN's  knowledge-base 
debugging  facility  includes  the  following  options: 


(1)a  trace  of  the  system's  "reasoning  process"  during  a  consultation; 
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•a  iatereetive  mechanism  for  reviewing  and  eor rooting  the  system's 
conclusions  is  generalisation  of  the  TBIRiSZAS  program ) ; 


(3)  sn  interface  to  the 


r*S  explanation  facility  to  produce 


automatically,  at  the  end  of  a  consultation,  explanations  of  how  the 
system  reached  Its  results; 

(4)  a  verification  mechanism  which  compares  the  system's  results  at  the 
end  of  a  consult  with  the  stored  "correct"  results  for  the  case  that 
were  saved  from  a  previous  interaction  with  the  TEIRESIAS-like 
option.  The  caaparieon  includes  explanations  of  why  the  system  made 
its  incorrect  conclusions  and  why  it  did  not  make  the  correct  ones. 


The  knowledge-base  debugging  tools  mentioned  above  allow  a  system 
builder  to  identify  problems  with  the  system's  knowledge  base  by  observing 
•rrors  in  It*  performance  on  test  oases.  While  thorough  testing  is  an 
essential  part  of  verifying  the  consistency  and  completeness  of  a  knowledge 
base,  it  is  rarely  possible  to  guarantee  that  a  knowledge-base  is  completely 
debugged,  even  after  hundreds  or  tost  runs. 

It  is  not  always  possible  to  tact  a  growing  knowledge  base  by  running 
Maple  oases .  TEHESXAS  was  developed  after  the  MYCIN  systea  was  fully 
functional  and  had  an  extensive  rule  sot.  EHTCXN  is  specifically  designed 
for  the  incremental  growth  of  a  knowledge  base  by  allowing  the  system  builder 
to  run  consultations  even  when  only  a  ekaletal  knowledge  base  has  been 
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defined.  The  task  of  building  an  EMYCIN  system  is  simply  to  encode  and  add 
the  knowledge.  In  contrast,  building  a  new  expert  system  typically  starts 
with  the  selection  of  knowledge  representation  formalisms  and  the  design  of  a 
program  to  use  the  knowledge.  Only  when  this  had  been  done  it  is  possible  to 
encode  the  knowledge  and  write  the  program  -  The  system  may  not  be  ready  to 
run  tests,  even  on  simple  cases,  until  the  entire  knowledge  base  is  encoded. 
When  an  expert  system  is  developed  in  this  manner,  it  would  be  convenient  if 
system  builders  could  run  a  preliminary  check  on  the  knowledge  base  before 
the  full  reasoning  mechanism  is  functioning  and  without  gathering  actual  data 
for  a  test  run. 


Knowledge-base  testing  tools,  therefore,  can  be  augmented  by  a 
program  which  systematically  checks  a  knowledge  base  for  campletoiess  and 
consistency.  This  checking  oan  be  done  during  the  system's  development,  even 
without  a  fully  functioning  reasoning  mechanism. 


ng  a  Rule-Based  System 


.3*1  Logical  Checks  for  Consistenc 


when  knowledge  is  represented  in  production  rules,  inconsistencies  in 
the  knowledge  base  appear  as: 


CONFLICT:  two  rules  succeed  in  the  same  situation  but  with 
conflicting  results. 


REDUNDANCY :  two  rules  succeed  in  the  same  situation  and  have  the  same 


results 
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SUBSUMPTION:  two  rules  have  the  same  results,  but  one  contains 
additional  restrictions  on  the  situations  in  which  it  will  succeed.  Whenever 
the  more  restrictive  rule  succeeds,  the  less  restrictive  rule  also  succeeds, 
resulting  in  redundancy. 

Conflict,  redundancy  and  subsunption  are  defined  above  as  logical 
conditions.  These  conditions  can  be  detected  if  syntax  allows  one  to  examine 
two  rules  and  determine  whether  situations  exist  in  which  both  can  succeed, 
and  whether  the  results  of  applying  the  two  rules  are  the  same,  conflicting, 
or  unrelated. 


4.3.2  Logical  Checks  for  Completeness 


Incompleteness  of  the  knowledge  base  is  the  result  of: 


MISSING  RULES:  a  situation  exists  in  which  a  particular  result  is 
required,  but  no  rule  can  succeed  in  that  situation  to  produce  the  desired 
result. 


Missing  rules  can  be  detected  logically  if  it  is  possible  to 
enumerate  all  circumstances  in  which  a  given  decision  should  be  made  or  a 
given  action  should  be  taken. 


4.3.3  Pragmatic  Considerations 


It  is  often  pragnatic  conditions,  not  purely  logical  ones,  that 
determine  whether  there  are  true  inconsistencies  in  a  knowledge  base.  The 
semantics  of  the  domain  may  modify  syntactic  analysis.  Of  the  three  types  of 
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inconsistency  described  above,  only  conflict  is  guaranteed  to  be  a  true 
error. 

In  practice,  logical  redundancy  nay  not  cause  problems.  In  a  system 
where  the  first  successful  rule  is  the  only  one  to  succeed,  a  problem  will 
arise  only  if  one  of  two  redundant  rules  is  revised  or  deleted  while  the 
other  is  left  unchanged.  On  the  other  hand,  in  a  system  using  a  scoring 
mechanism  (such  as  certainty  factors  in  EMYCIN  systems),  redundant  rules 
cause  the  same^toformation  to  be  counted  twice,  leading  to  erroneous 
increases  in  the  weight  of  their  conclusion. 

In  a  set  of  rules  that  accumulate  evidence  for  a  particular 
hypothesis,  one  rule  which  subsumes  another  may  cause  an  error  by  counting 
the  same  evidence  twice.  Alternatively,  the  expert  might  have  purposely 
written  the  rules  so  that  the  more  restrictive  one  adds  a  little  more  weight 
to  the  conclusion  nmde  by  the  less  restrictive  one. 

4 

An  exhaustive  syntactic  approach  for  Identifying  missing  rules  would 
assume  that  there  should  be  a  rule  which  applies  in  each  situation  defined  by 
all  possible  combinations  of  a  number  of  domain  variables.  Some  of  these 
combinations,  however,  might  not  be  meaningful.  As  with  consistency, 
checking  for  completeness  generally  requires  some  knowledge  of  the  problem 
domain. 

Because  of  these  pragsatlc  considerations,  an  automated  rule-checker 
should  display  potential  errors  and  allow  an  expert  to  indicate  which  ones 
represent  real  problems.  It  should  prompt  the  expert  for  domain-specific 
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information  to  explain  why  apparent  errors  are,  in  fact,  acceptable.  This 
information  should  be  represented  so  that  it  can  be  used  to  make  future 
checking  more  accurate. 


5  Rule-Checking  in  ONCOCIN 


5.1  Description  of  ONCOCIN 

ONCOCIN  is  a  rule-based  consultation  system  to  advise  physicians  at 
Stanford's  Oncology  Day  Care  Center  on  the  management  of  patients  who  are  on 
experimental  treatment  protocols.  These  protocols  serve  to  ensure  that  data 
from  patients  on  various  treatment  regimens  can  be  compared  to  evaluate  the 
success  of  therapy  and  to  assess  the  relative  effectiveness  of  alternative 
regimens.  A  protocol  specifies  when  the  patient  should  visit  the  clinic, 
what  chemotherapy  and/or  radiation  therapy  the  patient  should  receive  on  each 
visit,  when  laboratory  tests  should  be  performed,  and  under  what 
circumstances  and  in  what  ways  the  recommended  course  of  therapy  should  be 
modified. 


A  rule  in  ONCOCIN  is  a  production  with  an  ac tion  part  that  concludes 
a  value  for  some  parameter  on  the  basis  of  values  of  other  parameters  in  the 
rule's  condition  part.  Currently  all  parameter  values  can  be  determined  with 
oertalnty;  there  is  no  need  to  use  weighted  belief  measures.  When  a  rule 
sueoeeds,  its  action  parameter  becomes  known  so  no  other  rules  with  the  same 
action  parameter  will  be  tried. 
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Rules  specify  the  context  in  which  they  apply.  Ex  an  pies  of  ONCOCIN 
contexts  are  drugs,  chemotherapies  (i.e.,  drug  combinations),  and  protocols. 
A  rule  which  determines  the  dose  of  a  drug  may  be  specific  to  the  drug  alone, 
or  to  both  the  drug  and  the  ohemotherapy.  In  the  latter  case,  the  context  of 
the  rule  would  be  the  list  of  pairs  of  drug  and  chemotherapy  for  which  the 
rule  is  valid.  At  any  time  during  a  consultation,  the  current  context 
represents  the  particular  drug,  chemotherapy,  and  protocol  currently  under 
consideration. 

In  order  to  determine  the  value  of  a  parameter,  the  system  tries 
rules  which  conclude  about  that  parameter  and  which  apply  in  the  current 
context.  For  example,  Rule  75  shown  below  is  invoked  to  determine  the  value 
of  the  parameter  "current  attenuated  dose"  (point  a),  and  when  the  current 
context  is  a  drug  in  the  chemotherapy  MOPP,  or  a  drug  in  the  chemotherapy 
PAVe  (point  b). 

RULE  75 

[Action  Parameter]  (a)  To  determine  the  ourrent  attenuated  dose 

[Context]  (b)  for  all  drugs  in  MOPP,  or  for  all  drugs  in  PAVe: 

[Condition]  If:  1)  This  is  the  start  of  the  first  cycle 

after  a  cycle  was  aborted,  and 
2)  The  blood  counts  do  not  warrant  dose 
attenuation 

[Action]  Then:  Conclude  that  the  current  attenuated  dose 

is  75  percent  of  the  previous  dose. 

Certain  rules  for  determining  the  value  of  a  parameter  serve  special 
functions.  Some  give  a  "definitional"  value  in  the  specified  oontext.  These 
are  called  initial  rules  and  are  tried  first.  Other  rules  provide  a 
(possibly  context-dependent)  "default"  or  "usual"  value  in  the  event  that  no 
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other  rule  succeeded.  These  are  called  default  rules  and  are  applied  last. 
Rules  which  do  not  serve  either  of  these  special  functions  are  called  normal 
rules.  Concluding  a  parameter's  value,  then,  consists  of  trying,  in  order, 
three  groups  of  rules:  initial,  then  normal,  then  default.  A  rule's 
classification  tells  which  of  these  three  groups  it  belongs  to. 

Internally  in  LISP,  the  context,  condition,  action,  and 
classification  are  properties  of  an  atom  r< presenting  the  rule.  The  interned 
form  of  rule  75  is  shown  below. 

RULE075 

CONTEXT:  ((MOPP  DRUG) (PAVE  DRUG)) 

CONDITION:  (AND  ($IS  POST. ABORT  1) 

(SIS  NORMAL COUNTS  YES)) 

ACTION:  (CONCLUDEVALUE  ATTENDOSE  (PERCENTOF  75  ( PREVIOUSDOSE ) ) 

CLASSIFICATION:  NORMAL 

The  LISP  functions  which  are  used  in  conditions  or  actions  have 
tma plates  indicating  what  role  their  argunents  play.  For  exanple,  both  $IS 
and  CONCLUDEVALUE  take  a  parameter  as  their  first  argunent  and  a  value  of 
that  parameter  as  their  second  argument.  Each  function  also  has  a  descriptor 
representing  its  meaning.  For  example,  the  descriptor  of  $IS  shows  that  the 
function  will  suoceed  when  the  parameter  value  of  its  first  argunent  is  equal 
to  its  second  argunent. 
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possible  to  determine  the  combination  of  values  of  condition  parameters  which 
will  cause  a  rule  to  succeed.  The  rule's  context  property  shows  the 
context(s)  in  which  the  rule  applies.  The  context  and  condition  of  two  rules 
can  therefore  be  exasined  to  determine  if  there  are  situations  in  which  both 
can  succeed.  If  so,  and  the  rules  oonclude  different  values  for  the  same 
parameter,  they  are  in  conflict.  If  they  conclude  the  same  value  for  the 
same  parameter,  they  are  redundant.  If  they  are  the  same  except  that  one 
contains  extra  condition  clauses,  then  one  subsumes  the  other. 

These  definitions  of  inconsistencies  simplify  the  task  of  checking 
the  knowledge  base.  The  rules  can  be  partitioned  into  disjoint  sets,  each  of 
which  concludes  about  the  same  parameter  in  the  same  context.  The  resulting 

e 

rule  sets  can  be  checked  independently.  To  (heck  a  set  of  rules,  the 
program: 

(1)  finds  all  parameters  used  in  the  conditions  of  these  rules; 

(2)  makes  a  table,  displaying  all  possible  combinations  of  condition 
parameter  values  and  the  corresponding  values  which  will  be 
concluded  for  the  action  parameter1; 

(3)  checks  the  tables  for  conflict,  redundancy,  subsumption,  and  missing 
rules;  than  displays  the  table  with  a  sunmary  of  any  potential 


1 Because  a  parameter's  value  is  always  known  with  certainty  and  the 
possible  values  are  mutually  exclusive,  the  different  ccmbinations  of 
condition  parameter  values  are  disjoint.  If  a  rule  corresponding  to  one 
combination  succeeds,  rules  corresponding  to  other  combinations  in  the  same 
table  will  fail.  This  would  not  be  true  in  an  EMYCIN  consultation  system  in 
whioh  the  values  of  some  parameters  can  be  concluded  with  less  than  caaplete 
certainty.  In  suoh  cases,  the  combinations  in  a  given  table  would  not 
necessarily  be  disjoint. 
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errors  that  were  found.  The  rule  checker  assunes  that  there  should 
be  a  rule  for  each  possible  combination  of  values  of  condition 
parameters;  it  hypothesizes  missing  rules  based  on  this  assumption2. 

ONCOCIN 's  rule-ohecker  dynamically  examines  a  rule  set  to  determine 
which  condition  parameters  are  errantly  used  to  conclude  a  given  action 
parameter.  These  parameters  determine  what  columns  should  appear  in  the 
table  for  the  rule  set.  The  program  does  not  expect  that  each  of  the 
parameters  should  be  used  in  every  rule  in  the  set  (as  illustrated  in  by  rule 
76  in  the  exanple  below).  In  contrast,  TEIRESIAS  examined  the  "nearly 
complete"  MTCIN  knowledge  base  and  built  static  rule  models  showing  (among 
other  things)  which  condition  parameters  were  used  (in  the  existing  knowledge 
base)  to  conclude  a  given  action  parameter.  When  a  new  rule  was  added  to 
MTCIN,  it  was  compared  with  the  rule  model  for  its  action  parameter. 
TEIRESIAS  proposed  missing  clauses  if  some  condition  parameters  in  the  model 
did  not  appear  in  the  new  rule. 


5.3  An  Exmnple 


ONCOCIN' s  rule  checking  program  oan  check  the  entire  rule  base,  or 
can  interface  with  the  system’s  knowledge  acquisition  program  and  check  only 
those  rules  affected  by  recent  changes  to  the  knowledge  base.  This  latter 
mode  is  illustrated  by  the  example  in  Fig.  1;  the  systrnn  builder  is  trying  to 
determine  whether  the  recent  addition  of  one  rule  and  deletion  of  another 
have  introduced  errors. 

2We  plan  to  add  a  mechanism  to  acquire  information  about  the  meaning 
of  parameters  and  the  relationships  among  them,  and  to  use  this  information 
to  omit  semantically  Impossible  combinations  from  subsequent  tables. 
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The  rules  checked  in  the  exaeple  conclude  the  current  attenuated  dose 
for  the  drug  cytoxan  in  the  chemotherapy  CVP.  There  are  three  condition 
paraaeters  commonly  used  in  those  rules.  Of  these,  NORMAL COUNTS  takes  "YES" 
or  "NO"  as  its  value.  CYCLE  and  SIQXRT  take  integer  values.  The  only  value 
of  CYCLE  or  SIQXRT  which  was  Mentioned  explicitly  in  any  rule  is  "1"; 
therefore,  the  table  has  rows  for  values  "1”  and  "OTHER"  (i.e.,  other  than 
1). 

The  table  shows  that  rule  80  concludes  that  "attenuated  dose"  should 
have  the  value  "250  nilligraas  per  square  neter"  when  the  blood  counts  do  not 
warrant  dose  attenuation  (NORMAL COUNTS  »  YES),  the  chemotherapy  -cycle  nuaber 
is  1  (CYCLE  *  1),  and  this  is  the  first  cycle  after  significant  radiation 
( SIGXRT  s  1).  This  combination  of  values  of  the  condition  paraaeters  is 
labeled  Cl. 

Rule  76  can  succeed  in  the  same  situation  (Cl)  as  rule  80,  but  it 

4 

concludes  a  different  dose.  These  rules  do  not  conflict,  however,  because 
rule  76  is  a  "default"  rule  which  will  be  invoked  only  if  all  "normal"  rules 
(Including  rule  80)  fail.  Note  that  N0RMALC0UNTS  is  the  only  condition 
parameter  which  appears  explicitly  in  rule  76,  as  indicated  by  the 
parentheses  around  values  of  the  other  two  paraaeters.  Rule  76  will  succeed 
in  all  combination  which  include  NORMALCOUNTS  s  YES  (namely  Cl,  C3,  C5»  and 
C7). 

Rules  667  and  67  are  redundant  because  both  use  combination  C2  to 
conolude  the  value  labled  V2  (  250  mg/m2  attenuated  by  the  idnlmua  count 


attenuation) 
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Rule  600  la  in  oonfliet  with  rule  69  because  both  use  oanbination  C6, 
but  they  conclude  different  values  (and  both  are  categorized  as  "normal" 
rules). 

No  rules  exist  for  combinations  C4  and  C8,  so  the  program 
hypothesizes  that  rules  are  basing. 


Rule- Checking  in  ONCOCIN 


AAAI-82 


Rule  sett  667  600  82  80  69  67  76 
Contexts  the  drug  CYTOXAN  in  the  chemotherapy  CVP 
Action  Parameters  the  current  attenuated  doee 
Condition  Parameters s 

NORMAL COUNTS  -  the  blood  counts  do  not  warrant  dose  attenuation 

CYCLE  -  the  current  chemotherapy  cycle  number 

SIGXRT  -  the  nuaber  of  cycles  since  significant  radiation 

Values  too  long  to  appear  in  the  Value  ooluens 

VI  -  the  previous  dose  advanced  by  50  mg/r 
V2  -  250  mg/a r  attenuated  by  the  rtnlmua  count  attenuation 
V3  -  the  mlniatai  of  250  mg/af  and  the  previous  dose 
V4  -  the  adnimia  of  250  mg/a2  and  the  previous  dose  attenuated 
by  the  mLnimiai  count  attenuation 


Evaluation 

Rule 

Value 

NORMAL COUNTS 

CYCLE 

SIGXRT 

Combination 

80 

250tag/a2 

YES 

1 

1 

Cl 

76 

(D) 

VI 

YES 

(1) 

(1) 

Cl 

R 

667 

V2 

NO 

1 

1 

C2 

R 

67 

V2 

NO 

1 

1 

C2 

76 

(D) 

VI 

YES 

(1) 

(OTHER) 

C3 

M 

e 

NO 

1 

OTHER 

C*l 

82 

V3 

YES 

OTHER 

1 

C5 

76 

(D) 

VI 

YES 

(OTHER) 

(1) 

C5 

C 

600 

V3 

NO 

OTHER 

1 

C6 

r 

w 

69 

n 

NO 

OTHER 

1 

C6 

76 

(D) 

VI 

YES 

(OTHER) 

(OTHER) 

C7 

M 

e 

NO 

OTHER 

OTHER 

C8 

SUMMARY  OF  COMPARISON 

Conflict  exists  in  ccmbination(s)s  C6  (RULE600  RULE 06 9) 

Redundancy  exists  in  combination (s)s  C2  (RULE667  RULE067) 

Missing  rules  are  in  combination (s)s  C4,  C8 

NOTES 

Evaluations 

M  -  Missing}  C  -  Conflict;  R  -  Redundant. 

Rules s 

Default  rule  are  indicated  by  (D). 

Values  of  Condition  Parameters! 

A  value  in  parentheses  indicates  that  the  parameter  is  not  explicitly 
used  in  the  rule,  but  the  rule  will  sucoeed  when  parameter  has  this  value. 

Figure  2*  An  Bxmaple  of  the  Rule-Checking  Program 
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The  system  builder  can  enter  ONCOCIN's  knowledge  acquisition  program 
to  correct  any  of  the  errors  found  by  the  rule-checker.  A  missing  rule  can 
be  displayed  in  either  LISP  or  English  (Fig.  2),  and  added  to  the  system’s 
knowledge  base  after  the  expert  has  provided  a  value  for  its  action 
parameter. 


Missing  rule  corresponding  to  combination  C4: 

To  determine  the  current  attenuated  dose  for  Cytoxan  in  CVP: 

If:  1)  The  blood  counts  do  warrant  dose  attenuation, 

2)  The  current  chemotherapy  cycle  nunber  is  1,  and 

3)  This  is  not  the  start  of  the  first  cycle  after 
significant  radiation 

Then:  Conclude  that  the  current  attenuated  dose  is  . 

Figure  2.  Proposed  Missing  Rule  (English  Translation) 

Note  that  no  value  is  given  for  the  action  parameter;  this  could 
be  filled  in  by  the  system  builder  if  the  rule  looked  appropriate  for 
addition  to  the  knowledge  base. 


If  a  summary  table  is  too  big  to  display,  it  is  divided  into  a  nunber 
of  subtables  by  assigning  constant  values  to  some  of  the  condition 
parameters.  If  the  conditions  involve  ranges  of  numeric  values,  the  table 
will  displays  these  ranges  graphically  as  illustrated  in  Fig.  3. 
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Rule  sett  33  2* 

Context!  the  drug  OTIC  In  the  ah emo therapy  ABVD 
Aetlon  Parameters  the  does  ettenuetlon  due  to  low  HBC 


Default  veluet  100 


Evaluation  Rule 

Value 

(percentage) 

NBC  (in  thousands) 

0  1.5  2  3  5 

Combination 

33 

25 

....eeeao . 

Cl 

24 

50 

. •••0... 

C2 

SUMMARY  OF  COMPARISON 
No  probleas  were  found. 

NOTES 

•*s  appear  beneath  values  included  by  the  rule 
0's  appear  beneath  upper  or  lower  bounds  that 
are  not  included. 

E.g.,  Rule  33  applies  when  1.5  <=  NBC  <  2.0 

Figure  A  Table  of  Rules  with  Ranges  of  Nuaerical  Values 


5.4  Effects  of  the  Program 


The  rule  checking  program  described  in  this  paper  was  developed  at 
the  same  time  that  ONCOCIN* s  knowledge  base  was  being  built.  During  this 
time,  periodic  runs  of  the  rule  checker  suggested  missing  rules  that  had  been 
overlooked  by  the  oncology  expert.  It  also  detected  conflicting  and 
redundant  rules;  these  generally  arose  because  a  rule  had  the  incorrect 
oontext  and  therefore  appeared  in  the  wrong  table. 


A  nusber  of  inconsistencies  in  the  use  of  domain  concepts  were 
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revealed  by  the  rule  checker.  For  exam pie,  on  one  oocasion  the  program 
proposed  a  missing  rule  for  a  meaningless  combination  of  condition  parameter 
values.  In  discussing  the  domain  knowledge  that  expressed  the 

interrelationship  among  the  values,  it  became  clear  that  a  number  of 
individual  yes/no  valued  parameters  really  could  be  represented  more 
logically  as  different  values  for  the  same  parameter. 

The  knowledge  engineers  and  oncology  experts  alike  have  found  the 
rule  checker's  tabular  display  of  rule  sets  muoh  easier  to  interpret  than  a 
rule-by-rule  display.  Having  tabular  aiamariea  of  related  rules  has 
facilitated  the  task  of  modifying  the  knowledge  base. 

i 


6  Concluding  Remarks 

The  program  we  have  described  assists  a  knowledge  engineer  in 
ensuring  the  consistency  and  completeness  of  the  rule  set  in  the  ONCOCIN 
rule-based  consultation  system.  The  program  has  already  proved  useful  in 
development  of  that  systan.  The  program's  desipi  is  general  so  that  it  could 
be  adapted  to  other  rule-based  systems. 


8- 


v^n 


References 


References 


AAAI-82 


1.  Davis,  R.  Applications  of  Meta-level  Knowledge  to  the 

Construction,  Maintenance,  and  Use  of  Large  Knowledge  Bases.  Doctoral 
dissertation,  Computer  Science  Department,  Stanford  University,  June 
1976. 

2.  Shortliffe,  E.H.  Computer-Based  Medical  Consultations: 

MYCIN.  Elsevier/North  Holland  Publishing  Canpany,  New  York,  1976. 

3*  Shortliffe,  E.H. ,  Scott,  A.C. ,  Bischoff ,  M.B. ,  Campbell,  A.B. ,  van 
Melle,  W. ,  and  Jacobs,  C.D.  ONGOCIN:  An  expert  system  for  oncology 
protocol  management.  Proceedings  of  7th  IJCAI,  pp.  876- 
881,  Vancouver,  B.C.,  August  1981. 

M.  van  Melle,  N.  A  Domain-Independent  System  that  Aids  in 
Constructing  Knowledge-Based  Consultation  Programs .  Doctoral 


dissertation,  Computer  Science  Department,  Stanford  University,  June 


Distribution  Statement 


Defense  Documentation  Center 
Cameron  Station 
Alexandria,  VA  22314 

Office  of  Naval  Research 
Arlington,  VA  22217 

Information  Systems  Program  (437) 
Code  200 
Code  455 
Code  458 

Office  of  Naval  Research 
Caste rn/Central  Regional  Office 
Bldg.  114  Section  D 
666  Summer  St. 

Boston,  HA  02210 

Office  of  Naval  Research 
Branch  Office,  Chicago 
536  South  Clark  St. 

Chicago,  IL  60605 

Office  of  Naval  Research 
Western  Regional  Office 
1030  East  Green  St. 

Pasadena,  CA  91106 

Naval  Research  Laboratory 

Technical  Information  Division,  Code  2627 

Washington,  DC  20375 

Dr.  A.  L.  Slafkosky 
Scientific  Advisor 

Comnandant  of  the  Marine  Corps  (RD-1) 
Washington,  DC  20380 

Naval  Ocean  Systems  Center 
Advanced  Software  Technology  Division 
Code  5200 

San  Diego,  CA  92152 
Mr.  E.  H.  Glelssner 

Naval  Ship  Research  A  Development  Center 
Computation  and  Mathematics  Department 
Bethesda,  MD  20084 

Capt.  Grace  M.  Hopper  (008) 

Naval  Data  Automation  Conmand 
Washington  Navy  Yard 
Bldg.  166 

Washington,  DC  20374 

Captain  R.  Martin 
3311  West  Avenue 
Newport  News,  VA  23607 


12  copies 

2  copies 
1  copy 
1  copy 
1  copy 

1  copy 

1  copy 

1  copy 

6  copies 

1  copy 

1  copy 

1  copy 

1  copy 

1  copy 


